The present invention relates generally to data processing systems and more specifically to a personalized search engine that utilizes latent class models called aspect models or Probabilistic Latent Semantic Analysis or Indexing.
The rapid growth of digital data storage and the overwhelming supply of on-line information provided by today""s communication networks creates a risk of constant information overload. One of the key problems of modern information society is the increasing difficulty in accessing relevant information while suppressing the overwhelming mass of irrelevant data. Most importantly, the notion of relevance is highly individual and thus difficult to formalize in general terms. Information Filtering refers to the general problem of separating useful and important information from nuisance data. Individual users have different preferences, opinions, judgments, tastes, and cultural backgrounds. In order to support different individuals in their quest for information, an automated filtering system has to take into account the diversity of preferences and the inherent relativity of information value.
One commonly distinguishes between (at least) two major approaches to information filtering. The first approach is content-based filtering in which information organization is based on properties of the object or the carrier of information. The second approach is collaborative filtering (or social filtering), in which the preference-behavior and qualities of other persons are exploited in speculating about the preferences of a particular individual. Information Filtering technology had a huge impact on the development of the Internet and the e-commerce boom.
Search engines are classical content-based systems based on pattern matching technology to quickly retrieve information based on query keywords. Search engines are fast and have proven to be scalable to data sets of the size of the Internet, but they have inherent limitations. Due to the simple nature of direct pattern matching they provide no understanding of the sense of a word and the context of its use, often resulting in an unexpected variety of search results. These results are typically explained by word matching, but are meaningless on the intentional level. Search engines typically are unable to effectively process over-specific queries where keywords are closely related to actual document content, yet do not appear in the actual text. Search engines are also frequently non-personalized, i.e., they provide the same results, independent of the user history, and they have no effective mechanism to learn from user satisfaction.
At the other end of the spectrum, E-commerce sites use recommender systems to suggest products to their customers. The products can be recommended based on the top overall sellers on a site, or on the demographics of the customer, or an analysis of the past buying behavior of the customer as a prediction for future buying behavior. Recommender systems aim at personalization on the Web, enabling individual treatment of each customer. However, recommender systems and their underlying collaborative filtering techniques have several shortcomings. Poor predictions at best can be made for new objects and new users (xe2x80x9cEarly rater problemxe2x80x9d). In many domains, only a small percentage of possible items receive ratings for a given user (xe2x80x9cScarcity problemxe2x80x9d). There are often users with different interests and tastes than any other users in the database (xe2x80x9cGrey sheep problemxe2x80x9d). More fundamentally, while many users may share some common interest, it may be extremely difficult to find a sufficient number of users that share all interests.
Therefore, in light of the foregoing deficiencies in the prior art, the applicant""s invention is herein presented.
The disclosed system provides a method for the personalized filtering of information and the automated generation of user-specific recommendations. The system goes through 3 phases: 1) Information Gathering, 2) System Learning, and 3) Information Retrieval. The disclosed system is concerned primarily with the final two phases (System Learning and Information Retrieval). In the Information Gathering phase, information about the data to be retrieved (DOCUMENT DATA) and about the individual users (USER DATA) is collected. The USER DATA can be gathered explicitly through questionnaires, etc. or can be implied though observing user behavior such as Internet history logs, demographic information, or any other relevant sources of information. The DOCUMENT DATA can be gathered though a variety of methods including Internet crawling, topic taxonomies or any other relevant source of information. Once the Information Gathering phase is completed, the System Learning phase is initiated. The system employs a statistical algorithm that uses available USER DATA and DOCUMENT DATA to create a statistical latent class model (MODEL), also known as Probabilistic Latent Semantic Analysis (PLSA). The system learns one or more MODELS based on the USER DATA, DOCUMENT DATA, and the available database containing data obtained from other users. Within the MODEL, probabilities for words extracted from training data are calculated and stored in at least one matrix. An extended inverted index may also be generated and stored along with the MODEL in order to facilitate more efficient information gathering. The MODEL may be used in other applications such as the unsupervised learning of topic hierarchies and for other data mining functions such as identifying user communities. Various parts of the Information Gathering phase and the System Learning phase are repeated from time to time in order to further refine or update the MODEL. This refined or updated model will result in even higher levels of accuracy in processing the user""s query. The final phase is the Information Retrieval phase. The user may enter a query. Once the query is entered into the system, the MODEL is utilized in calculating probabilities for every word in a document based upon at least 1) the user query, or 2) words associated with the users query in the MODEL, or 3) document information. All of the probabilities for a given document are added together yielding a total relevance xe2x80x9cscorexe2x80x9d after which related documents are compared using this relevance score. The results are returned in descending order of relevance organized into at least one result list. Through the use of the MODEL, the system provides two benefits to the user: 1) the search results are personalized as each MODEL may be created in part using USER DATA, and 2) results for new users are somewhat personalized from the initial use through collaborative filtering based upon USER DATA for other system users.